Pandas Profiling Report

Dataset statistics

Number of variables	12
Number of observations	550068
Missing cells	556885
Missing cells (%)	8.4%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	193.8 MiB
Average record size in memory	369.4 B

Variable types

NUM	6
CAT	5
BOOL	1

Reproduction

Analysis started	2020-07-03 09:40:06.074475
Analysis finished	2020-07-03 10:19:38.180536
Version	pandas-profiling v2.6.0
Command line	`pandas_profiling --config_file config.yaml [YOUR_FILE.csv]`
Download configuration	config.yaml

`Product_ID` has a high cardinality: 3631 distinct values	High cardinality
`Product_Category_2` has 173638 (31.6%) missing values	Missing
`Product_Category_3` has 383247 (69.7%) missing values	Missing
`Occupation` has 69638 (12.7%) zeros	Zeros

User_ID
Real number (ℝ_≥0)

Distinct count	5891
Unique (%)	1.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1003028.842
Minimum	1000001
Maximum	1006040
Zeros	0
Zeros (%)	0.0%
Memory size	4.2 MiB

Quantile statistics

Minimum	1000001
5-th percentile	1000329
Q1	1001516
median	1003077
Q3	1004478
95-th percentile	1005747
Maximum	1006040
Range	6039
Interquartile range (IQR)	2962

Descriptive statistics

Standard deviation	1727.591586
Coefficient of variation (CV)	0.001722374784
Kurtosis	-1.195500781
Mean	1003028.842
Median Absolute Deviation (MAD)	1499.97937
Skewness	0.003065551851
Sum	5.517340693e+11
Variance	2984572.686

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[1000001. 1000002.5 1000004.5 1000005.5 1000006.5 ... 1006036.5 1006037.5 1006038.5 1006039.5 1006040. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
1001680	1026	0.2%
1004277	979	0.2%
1001941	898	0.2%
1001181	862	0.2%
1000889	823	0.1%
1003618	767	0.1%
1001150	752	0.1%
1001015	740	0.1%
1005795	729	0.1%
1005831	727	0.1%
Other values (5881)	541765	98.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1000001	35	< 0.1%
1000002	77	< 0.1%
1000003	29	< 0.1%
1000004	14	< 0.1%
1000005	106	< 0.1%

Value	Count	Frequency (%)
1006040	180	< 0.1%
1006039	74	< 0.1%
1006038	12	< 0.1%
1006037	122	< 0.1%
1006036	514	0.1%

Product_ID
Categorical

HIGH CARDINALITY

Distinct count	3631
Unique (%)	0.7%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

P00265242	1880
P00025442	1615
P00110742	1612
P00112142	1562
P00057642	1470
Other values (3626)	541929

Value	Count	Frequency (%)
P00265242	1880	0.3%
P00025442	1615	0.3%
P00110742	1612	0.3%
P00112142	1562	0.3%
P00057642	1470	0.3%
P00184942	1440	0.3%
P00046742	1438	0.3%
P00058042	1422	0.3%
P00145042	1406	0.3%
P00059442	1406	0.3%
Other values (3621)	534817	97.2%

Length

Max length	9
Mean length	8.982729408
Min length	8

Value	Count	Frequency (%)
Decimal_Number	10	90.9%
Uppercase_Letter	1	9.1%

Value	Count	Frequency (%)
Common	10	90.9%
Latin	1	9.1%

Value	Count	Frequency (%)
ASCII	11	100.0%

Gender
Categorical

Distinct count	2
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

M	414259
F	135809

Value	Count	Frequency (%)
M	414259	75.3%
F	135809	24.7%

Length

Max length	1
Mean length	1
Min length	1

Value	Count	Frequency (%)
Uppercase_Letter	2	100.0%

Value	Count	Frequency (%)
Latin	2	100.0%

Value	Count	Frequency (%)
ASCII	2	100.0%

Age
Categorical

Distinct count	7
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

26-35	219587
36-45	110013
18-25	99660
46-50	45701
51-55	38501
Other values (2)	36606

Value	Count	Frequency (%)
26-35	219587	39.9%
36-45	110013	20.0%
18-25	99660	18.1%
46-50	45701	8.3%
51-55	38501	7.0%
55+	21504	3.9%
0-17	15102	2.7%

Length

Max length	5
Mean length	4.894358516
Min length	3

Value	Count	Frequency (%)
Decimal_Number	9	81.8%
Dash_Punctuation	1	9.1%
Math_Symbol	1	9.1%

Value	Count	Frequency (%)
Common	11	100.0%

Value	Count	Frequency (%)
ASCII	11	100.0%

Occupation
Real number (ℝ_≥0)

ZEROS

Distinct count	21
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	8.07670688
Minimum	0
Maximum	20
Zeros	69638
Zeros (%)	12.7%
Memory size	4.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	2
median	7
Q3	14
95-th percentile	20
Maximum	20
Range	20
Interquartile range (IQR)	12

Descriptive statistics

Standard deviation	6.522660487
Coefficient of variation (CV)	0.8075891059
Kurtosis	-1.216113649
Mean	8.07670688
Median Absolute Deviation (MAD)	5.772156967
Skewness	0.4001401099
Sum	4442738
Variance	42.54509983

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[ 0. 1.5 2.5 3.5 4.5 ... 16.5 17.5 18.5 19.5 20. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
4	72308	13.1%
0	69638	12.7%
7	59133	10.8%
1	47426	8.6%
17	40043	7.3%
20	33562	6.1%
12	31179	5.7%
14	27309	5.0%
2	26588	4.8%
16	25371	4.6%
Other values (11)	117511	21.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	69638	12.7%
1	47426	8.6%
2	26588	4.8%
3	17650	3.2%
4	72308	13.1%

Value	Count	Frequency (%)
20	33562	6.1%
19	8461	1.5%
18	6622	1.2%
17	40043	7.3%
16	25371	4.6%

City_Category
Categorical

Distinct count	3
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

B	231173
C	171175
A	147720

Value	Count	Frequency (%)
B	231173	42.0%
C	171175	31.1%
A	147720	26.9%

Length

Max length	1
Mean length	1
Min length	1

Value	Count	Frequency (%)
Uppercase_Letter	3	100.0%

Value	Count	Frequency (%)
Latin	3	100.0%

Value	Count	Frequency (%)
ASCII	3	100.0%

Stay_In_Current_City_Years
Categorical

Distinct count	5
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

1	193821
2	101838
3	95285
4+	84726
0	74398

Value	Count	Frequency (%)
1	193821	35.2%
2	101838	18.5%
3	95285	17.3%
4+	84726	15.4%
0	74398	13.5%

Length

Max length	2
Mean length	1.154028229
Min length	1

Value	Count	Frequency (%)
Decimal_Number	5	83.3%
Math_Symbol	1	16.7%

Value	Count	Frequency (%)
Common	6	100.0%

Value	Count	Frequency (%)
ASCII	6	100.0%

Marital_Status
Boolean

Distinct count	2
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.2 MiB

0	324731
1	225337

Frequency Table

Value	Count	Frequency (%)
0	324731	59.0%
1	225337	41.0%

Product_Category_1
Real number (ℝ_≥0)

Distinct count	20
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5.404270018
Minimum	1
Maximum	20
Zeros	0
Zeros (%)	0.0%
Memory size	4.2 MiB

Quantile statistics

Minimum	1
5-th percentile	1
Q1	1
median	5
Q3	8
95-th percentile	13
Maximum	20
Range	19
Interquartile range (IQR)	7

Descriptive statistics

Standard deviation	3.936211369
Coefficient of variation (CV)	0.7283520913
Kurtosis	1.234756972
Mean	5.404270018
Median Absolute Deviation (MAD)	3.001889578
Skewness	1.025734934
Sum	2972716
Variance	15.49375994

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[ 1. 2.5 3.5 4.5 5.5 ... 16.5 17.5 18.5 19.5 20. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
5	150933	27.4%
1	140378	25.5%
8	113925	20.7%
11	24287	4.4%
2	23864	4.3%
6	20466	3.7%
3	20213	3.7%
4	11753	2.1%
16	9828	1.8%
15	6290	1.1%
Other values (10)	28131	5.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	140378	25.5%
2	23864	4.3%
3	20213	3.7%
4	11753	2.1%
5	150933	27.4%

Value	Count	Frequency (%)
20	2550	0.5%
19	1603	0.3%
18	3125	0.6%
17	578	0.1%
16	9828	1.8%

Product_Category_2
Real number (ℝ_≥0)

MISSING

Distinct count	17
Unique (%)	< 0.1%
Missing	173638
Missing (%)	31.6%
Infinite	0
Infinite (%)	0.0%

Mean	9.842329251
Minimum	2
Maximum	18
Zeros	0
Zeros (%)	0.0%
Memory size	4.2 MiB

Quantile statistics

Minimum	2
5-th percentile	2
Q1	5
median	9
Q3	15
95-th percentile	16
Maximum	18
Range	16
Interquartile range (IQR)	10

Descriptive statistics

Standard deviation	5.086589649
Coefficient of variation (CV)	0.5168075075
Kurtosis	-1.432266899
Mean	9.842329251
Median Absolute Deviation (MAD)	4.625958938
Skewness	-0.1627577144
Sum	3704948
Variance	25.87339425

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
8	64088	11.7%
14	55108	10.0%
2	49217	8.9%
16	43255	7.9%
15	37855	6.9%
5	26235	4.8%
4	25677	4.7%
6	16466	3.0%
11	14134	2.6%
17	13320	2.4%
Other values (7)	31075	5.6%
(Missing)	173638	31.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
2	49217	8.9%
3	2884	0.5%
4	25677	4.7%
5	26235	4.8%
6	16466	3.0%

Value	Count	Frequency (%)
18	2770	0.5%
17	13320	2.4%
16	43255	7.9%
15	37855	6.9%
14	55108	10.0%

Product_Category_3
Real number (ℝ_≥0)

MISSING

Distinct count	15
Unique (%)	< 0.1%
Missing	383247
Missing (%)	69.7%
Infinite	0
Infinite (%)	0.0%

Mean	12.66824321
Minimum	3
Maximum	18
Zeros	0
Zeros (%)	0.0%
Memory size	4.2 MiB

Quantile statistics

Minimum	3
5-th percentile	5
Q1	9
median	14
Q3	16
95-th percentile	17
Maximum	18
Range	15
Interquartile range (IQR)	7

Descriptive statistics

Standard deviation	4.125337632
Coefficient of variation (CV)	0.325644019
Kurtosis	-0.8080661151
Mean	12.66824321
Median Absolute Deviation (MAD)	3.565943094
Skewness	-0.7654458894
Sum	2113329
Variance	17.01841057

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
16	32636	5.9%
15	28013	5.1%
14	18428	3.4%
17	16702	3.0%
5	16658	3.0%
8	12562	2.3%
9	11579	2.1%
12	9246	1.7%
13	5459	1.0%
6	4890	0.9%
Other values (5)	10648	1.9%
(Missing)	383247	69.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
3	613	0.1%
4	1875	0.3%
5	16658	3.0%
6	4890	0.9%
8	12562	2.3%

Value	Count	Frequency (%)
18	4629	0.8%
17	16702	3.0%
16	32636	5.9%
15	28013	5.1%
14	18428	3.4%

Purchase
Real number (ℝ_≥0)

Distinct count	18105
Unique (%)	3.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	9263.968713
Minimum	12
Maximum	23961
Zeros	0
Zeros (%)	0.0%
Memory size	4.2 MiB

Quantile statistics

Minimum	12
5-th percentile	1984
Q1	5823
median	8047
Q3	12054
95-th percentile	19336
Maximum	23961
Range	23949
Interquartile range (IQR)	6231

Descriptive statistics

Standard deviation	5023.065394
Coefficient of variation (CV)	0.5422152805
Kurtosis	-0.3383775656
Mean	9263.968713
Median Absolute Deviation (MAD)	4069.959166
Skewness	0.6001400037
Sum	5095812742
Variance	25231185.95

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[1.20000e+01 1.25000e+01 1.35000e+01 2.45000e+01 2.55000e+01 ... 2.10805e+04 2.15685e+04 2.26535e+04 2.30405e+04 2.39610e+04], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
7011	191	< 0.1%
7193	188	< 0.1%
6855	187	< 0.1%
6891	184	< 0.1%
6960	183	< 0.1%
7012	183	< 0.1%
6879	182	< 0.1%
7166	182	< 0.1%
7027	182	< 0.1%
7165	180	< 0.1%
Other values (18095)	548226	99.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
12	101	< 0.1%
13	106	< 0.1%
14	95	< 0.1%
24	118	< 0.1%
25	113	< 0.1%

Value	Count	Frequency (%)
23961	3	< 0.1%
23960	4	< 0.1%
23959	2	< 0.1%
23958	4	< 0.1%
23956	1	< 0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

First rows

	User_ID	Product_ID	Gender	Age	Occupation	City_Category	Stay_In_Current_City_Years	Marital_Status	Product_Category_1	Product_Category_2	Product_Category_3	Purchase
0	1000001	P00069042	F	0-17	10	A	2	0	3	NaN	NaN	8370
1	1000001	P00248942	F	0-17	10	A	2	0	1	6.0	14.0	15200
2	1000001	P00087842	F	0-17	10	A	2	0	12	NaN	NaN	1422
3	1000001	P00085442	F	0-17	10	A	2	0	12	14.0	NaN	1057
4	1000002	P00285442	M	55+	16	C	4+	0	8	NaN	NaN	7969
5	1000003	P00193542	M	26-35	15	A	3	0	1	2.0	NaN	15227
6	1000004	P00184942	M	46-50	7	B	2	1	1	8.0	17.0	19215
7	1000004	P00346142	M	46-50	7	B	2	1	1	15.0	NaN	15854
8	1000004	P0097242	M	46-50	7	B	2	1	1	16.0	NaN	15686
9	1000005	P00274942	M	26-35	20	A	1	1	8	NaN	NaN	7871

Last rows

	User_ID	Product_ID	Gender	Age	Occupation	City_Category	Stay_In_Current_City_Years	Marital_Status	Product_Category_1	Product_Category_2	Product_Category_3	Purchase
550058	1006024	P00372445	M	26-35	12	A	0	1	20	NaN	NaN	121
550059	1006025	P00370853	F	26-35	1	B	1	0	19	NaN	NaN	48
550060	1006026	P00371644	M	36-45	6	C	1	1	20	NaN	NaN	494
550061	1006029	P00372445	F	26-35	1	C	1	1	20	NaN	NaN	599
550062	1006032	P00372445	M	46-50	7	A	3	0	20	NaN	NaN	473
550063	1006033	P00372445	M	51-55	13	B	1	1	20	NaN	NaN	368
550064	1006035	P00375436	F	26-35	1	C	3	0	20	NaN	NaN	371
550065	1006036	P00375436	F	26-35	15	B	4+	1	20	NaN	NaN	137
550066	1006038	P00375436	F	55+	1	C	2	0	20	NaN	NaN	365
550067	1006039	P00371644	F	46-50	0	B	4+	1	20	NaN	NaN	490

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Missing values

Sample

First rows

Last rows